Time Series and Data Management
================================

Dates
------

In the RISE toolbox, dates play a crucial role in representing
time series data. Dates can be either explicit, where each
observation is associated with a specific date, or undated, where
dates are simply represented as a sequence of integers.

To handle dates efficiently, RISE provides an abstract class
called ``rise_dates.dates``, from which various derived classes
inherit. These derived classes include:

* ``rise_dates.DailyDate``     (shortcut: ``rd``)
* ``rise_dates.WeeklyDate``    (shortcut: ``rw``)
* ``rise_dates.MonthlyDate``   (shortcut: ``rm``)
* ``rise_dates.QuarterlyDate`` (shortcut: ``rq``)
* ``rise_dates.HalfYearlyDate`` (shortcut: ``rh``)
* ``rise_dates.YearlyDate``    (shortcuts: ``ry`` or ``ra``)

Each derived class represents a specific frequency.

Base Class Methods
~~~~~~~~~~~~~~~~~~~

The base class ``rise_dates.dates`` provides several methods that
can be used with any derived class:

* ``char()`` -- converts dates to character representations.
* ``convert()`` -- converts dates to another frequency.
* ``datestr()`` -- returns a string representation of dates.
* ``eq()`` -- equality check.
* ``gt()``, ``ge()``, ``lt()``, ``le()`` -- comparisons.
* ``ismember()`` -- membership test.
* ``min()``, ``max()`` -- minimum / maximum date.
* ``ne()`` -- inequality check.
* ``unique()`` -- unique dates in a sequence.
* ``colon()`` -- creates a sequence of dates.
* ``double()`` -- converts dates to double format.
* ``intersect()`` -- intersection of two sets of dates.
* ``minus()``, ``plus()`` -- date arithmetic.
* ``ymd()`` -- extracts year, month, day components.

When passing a date to an estimation option (``estim_start_date``,
``estim_end_date``, ...), use ``date2serial`` to convert the char
form to the numeric / ``rise_dates.dates`` form the modern
validators accept::

    estim_start_date = date2serial('1990Q1')


Help Text for Derived Classes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. _rd:

Daily dates: ``rd``
""""""""""""""""""""

.. include:: rd_help.rst

.. _rw:

Weekly dates: ``rw``
""""""""""""""""""""

.. include:: rw_help.rst

.. _rm:

Monthly dates: ``rm``
""""""""""""""""""""""

.. include:: rm_help.rst

.. _rq:

Quarterly dates: ``rq``
"""""""""""""""""""""""

.. include:: rq_help.rst

.. _rh:

Half-yearly dates: ``rh``
""""""""""""""""""""""""""

.. include:: rh_help.rst

.. _ry:

Yearly dates: ``ry``
"""""""""""""""""""""

.. include:: ry_help.rst

.. _ra:

Annual dates: ``ra``
"""""""""""""""""""""

.. include:: ra_help.rst


Time Series
------------

This section provides information about time series and their
usage in the context of the RISE toolbox.

Time series data in essence is just a matrix, so in theory one
can just use matrices to represent the data. However, all
practitioners will share their frustration in trying to remember
the index location of the GDP and the annoyance in trying to
find out what ``data(:, 5)`` written by a co-worker represents.
The RISE toolbox addresses this problem by providing a time
series object that uses natural indices, improving readability
and reducing errors.

To use the time series object, initialize it with::

    db = ts('1990q1', randn(10, 4), {'gdp','r','wage','capital'})

The first argument specifies the starting date of the time series
and its frequency (here, 1990Q1). The second argument is the data
matrix, where each row represents a time period and each column
represents a variable. The third argument provides the natural
names of the variables for intuitive indexing.

If you want to provide descriptions or comments for each variable::

    db = ts('1990q1', randn(10, 2), {'gdp','pi'}, {'real gdp','inflation rate'})

You can display the time series with::

    disp(db)

Alternatively, set variable names and descriptions with ``set``::

    set(db, 'varnames', {'gdp','pi'}, 'description', {'real gdp','inflation rate'})


.. _ts-indexing:

Indexing and assignment
~~~~~~~~~~~~~~~~~~~~~~~~

The indexing surface of the time series object mirrors the
built-in ``table`` / ``timetable`` conventions, so that users
familiar with those classes feel at home, and the distinction
between "sub-time-series" and "raw values" is explicit.

.. list-table::
   :header-rows: 1
   :widths: 30 70

   * - Expression
     - Result
   * - ``db.PROP``
     - a property of the ts (``data``, ``dates``, ``frequency``, ``NumberOfObservations``, ...).
   * - ``db.VARNAME``
     - if ``VARNAME`` matches a variable name (and is not a property/method), returns the single-variable ts.
   * - ``db(I [, J [, K]])``
     - a **sub-time-series** (returns a ts). ``I``, ``J``, ``K`` are dates / date strings / logical / cellstr / positions / ``:`` / function-handle filters on pages.
   * - ``db{I [, J [, K]]}``
     - the **raw numeric block** at that selection (a double array of size ``nobs x nvars x npages``). Use this when the next step is plain MATLAB arithmetic; otherwise stay in the ts world with ``()``.
   * - ``db{-k}`` / ``db(-k)``
     - (dated ts only, ``k`` a scalar integer) shortcut for ``lag(db, k)``; returns a ts. ``db{+k}`` / ``db(+k)`` is ``lead(db, k)``. ``db{0}`` is ``db``. On undated ts a scalar numeric is the implicit date value instead.
   * - ``db.VARNAME = rhs``
     - replace one variable's column from a ts or a numeric column.
   * - ``db(I, ...) = rhs``
     - replace a sub-time-series (rhs ts or numeric).
   * - ``db{I, ...} = rhs``
     - raw numeric block write.

Scalar-numeric assignment (``db{-1} = X``, ``db(-1) = X``) is
**not** supported. The time-shift shortcut is read-only; build the
shifted ts first and assign with a date or logical selector if
you need to write into it.

Examples::

    db = ts('1990q1', randn(10, 4), {'gdp','r','wage','capital'});

    % Access values
    db.wage                          % single-variable ts
    db('wage')                       % same, sub-ts form
    db{'wage'}                       % raw 10x1 double array

    % Slice rows and columns
    db('1991q2', 'wage')             % single observation, as a ts
    db{'1991q2', 'wage'}             % the same value as a 1x1 double
    db('1990q2:1991q2', {'gdp','wage'})   % sub-ts (5 rows, 2 vars)
    db{:, {'gdp','wage'}}            % full sample, two vars, as a 10x2 double

    % Whole-year selection works across frequencies
    db(ry(1991))                     % the 4 quarters of 1991, as a ts

    % Underlying data, when you need plain MATLAB
    values(db)                       % equivalent to db.data

Time shifts (lags and leads) are expressed either with the
explicit methods ``lag(db, k)``, ``lead(db, k)``, ``shift(db, k)``
or, on a dated ts, with the brace/paren shortcut. Both preserve
the date axis -- the leading or trailing ``k`` observations are
NaN-padded so that the length of the output equals the length of
the input. The two forms are equivalent on a dated ts::

    growth = 100*(db.gdp / lag(db.gdp, 1) - 1);   % q/q growth, % terms
    growth = 100*(db.gdp / db.gdp{-1}      - 1);  % same thing, shorter

On an undated ts the shortcut is unavailable (a scalar numeric in
the braces is treated as the date value itself); use ``lag``,
``lead``, ``shift`` explicitly there.

The time series object seamlessly integrates with other parts of
the RISE toolbox, enabling you to perform VAR and DSGE analyses
effortlessly.


Data Cleaning
~~~~~~~~~~~~~~

* ``reset_data`` -- resets the data in the time series object.
* ``reset_start_date`` -- resets the start date.
* ``interpolate`` -- fills in missing data.
* ``aggregate`` -- aggregates data to a coarser frequency.
* ``dummy`` -- creates a dummy variable.
* ``step_dummy`` -- creates a step-dummy variable.
* ``cat`` -- joins multiple time series objects.


Time Series Transformations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* ``hpfilter`` -- the Hodrick-Prescott filter.
* ``ma_filter`` -- a moving-average filter.
* ``pdecomp`` -- parametric trend-seasonal-residual decomposition.
* ``npdecomp`` -- non-parametric trend-seasonal-residual
  decomposition.
* ``transform`` -- standard time-series transformations (level,
  growth, difference, log change, year-over-year variants),
  following the Haver mnemonics.


Time shifts
~~~~~~~~~~~~

These methods preserve the date axis -- the leading or trailing
``k`` rows are NaN-padded so that ``length(out) == length(db)``:

* ``lag(db, k)``   -- lag by ``k``. ``out.data(t,:,:) = db.data(t-k,:,:)``.
* ``lead(db, k)``  -- lead by ``k``. ``out.data(t,:,:) = db.data(t+k,:,:)``.
* ``shift(db, k)`` -- generic shift. Positive ``k`` is a lead,
  negative is a lag.

On a dated ts these are also available as the shortcut
``db{-k}`` / ``db(-k)`` (lag) and ``db{+k}`` / ``db(+k)`` (lead) --
see *Indexing and assignment* above.


Statistical Analysis
~~~~~~~~~~~~~~~~~~~~~

* ``regress`` -- OLS regression on the time series data.
* ``chowlin`` -- temporal disaggregation via Chow-Lin
  (interpolates a low-frequency series to high frequency using
  one or more related indicators).
* ``jbtest`` -- Jarque-Bera test.


Plotting tools for time series
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* ``bar`` -- bar plot.
* ``fanchart`` -- fan chart.
* ``plot`` -- plot the time series data.
* ``plot_real_time`` -- plot in real-time-conditioning format.
* ``plotyy`` -- plot with two separate axes.

See :doc:`../Plotting tools` for the broader plotting surface.